<!DOCTYPE html>
<html lang="zh-hans">

<head>
    
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no" />
<meta name="HandheldFriendly" content="True" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta http-equiv="Cache-Control" content="no-transform" />
<meta http-equiv="Cache-Control" content="no-siteapp" />
<meta name="generator" content="Hugo 0.109.0">


<link rel="shortcut icon" href="https://cdn.jsdelivr.net/gh/dsrkafuu/dsr-cdn-main@1/images/favicons/dsrca.ico" />



<title>前向传播、反向传播与感知器 - OffSummer</title>


<meta name="author" content="RQY" />


<meta name="description" content="A minimal Hugo theme with nice theme color." />


<meta name="keywords" content="mindspore, DL" />


<meta property="og:title" content="前向传播、反向传播与感知器" />
<meta name="twitter:title" content="前向传播、反向传播与感知器" />
<meta property="og:type" content="article" />
<meta property="og:url" content="/post/00-forward_backward_and_mlp/" /><meta property="og:description" content="设计思路
前向传播
前向传播是计算输出值$\hat y=W^Tx&#43;b$与损失函数$Loss(\hat y,y)$。具体而言就在神经网络中是根据输入的数据x与网络的权重w经过激活函数后输出$\hat y$，并根据$\hat y$与$y$计算出损失函数的过程。
反向传播
方向传播是计算损失函数的梯度，即$\Delta w=\alpha\frac{\partial L}{\partial w}$，$\Delta b=\alpha\frac{\partial L}{\partial b}$，$w = w - \Delta w $，$b = b - \Delta b $，其中$\alpha$是学习率。根据微积分中的梯度方向是函数值增长方向的原理，通过向梯度方向的反方向更新权重与偏置，来使得损失函数的取值最小化的过程。" />
<meta name="twitter:description" content="设计思路
前向传播
前向传播是计算输出值$\hat y=W^Tx&#43;b$与损失函数$Loss(\hat y,y)$。具体而言就在神经网络中是根据输入的数据x与网络的权重w经过激活函数后输出$\hat y$，并根据$\hat y$与$y$计算出损失函数的过程。
反向传播
方向传播是计算损失函数的梯度，即$\Delta w=\alpha\frac{\partial L}{\partial w}$，$\Delta b=\alpha\frac{\partial L}{\partial b}$，$w = w - \Delta w $，$b = b - \Delta b $，其中$\alpha$是学习率。根据微积分中的梯度方向是函数值增长方向的原理，通过向梯度方向的反方向更新权重与偏置，来使得损失函数的取值最小化的过程。" /><meta property="og:image" content="/img/og.png" />
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:image" content="/img/og.png" /><meta property="article:published_time" content="2022-04-10T13:41:20+08:00" /><meta property="article:modified_time" content="2022-04-10T13:41:20+08:00" />


<style>
    @media (prefers-color-scheme: dark) {
        body[data-theme='auto'] img {
            filter: brightness(60%);
        }
    }

    body[data-theme='dark'] img {
        filter: brightness(60%);
    }
</style>




<link rel="stylesheet" href="/assets/css/fuji.min.b4a21b5d3eb1d0a51297e31230a65fc25e387843e45ec3a2d9176cd8d163c216d99b9b13a618b28f537c3b559ec8a408183b0fbfad48daddb9befa7d3ef90eed.css" integrity="sha512-tKIbXT6x0KUSl&#43;MSMKZfwl44eEPkXsOi2Rds2NFjwhbZm5sTphiyj1N8O1WeyKQIGDsPv61I2t25vvp9PvkO7Q==" />








</head>

<body
  data-theme="auto"
  data-theme-auto='true'
  >
    <script data-cfasync="false">
  
  var fujiThemeData = localStorage.getItem('fuji_data-theme');
  
  if (!fujiThemeData) {
    localStorage.setItem('fuji_data-theme', 'auto');
  } else {
    
    if (fujiThemeData !== 'auto') {
      document.body.setAttribute('data-theme', fujiThemeData === 'dark' ? 'dark' : 'light');
    }
  }
</script>

    <header>
    <div class="container-lg clearfix">
        <div class="col-12 header">
            <a class="title-main" href="/">OffSummer</a>
            
            <span class="title-sub">Summer is going, but autumn does not come yet.</span>
            
        </div>
    </div>
</header>

    <main>
        <div class="container-lg clearfix">
            
            <div class="col-12 col-md-9 float-left content">
                
<article>
    
    <h2 class="post-item post-title">
        <a href="/post/00-forward_backward_and_mlp/">前向传播、反向传播与感知器</a>
    </h2>
    <div class="post-item post-meta">
        <span><i class="iconfont icon-today-sharp"></i>&nbsp;2022-04-10</span>

<span><i class="iconfont icon-file-tray-sharp"></i>&nbsp;725 字</span>
<span><i class="iconfont icon-time-sharp"></i>&nbsp;2 分钟</span>
<span><i class="iconfont icon-pricetags-sharp"></i>&nbsp;<a href="/tags/mindspore">mindspore</a>&nbsp;<a href="/tags/dl">DL</a>&nbsp;</span>

    </div>
    
    <div class="post-content markdown-body">
        <h2 id="设计思路">设计思路</h2>
<h3 id="前向传播">前向传播</h3>
<p>前向传播是计算输出值$\hat y=W^Tx+b$与损失函数$Loss(\hat y,y)$。具体而言就在神经网络中是根据输入的数据x与网络的权重w经过激活函数后输出$\hat y$，并根据$\hat y$与$y$计算出损失函数的过程。</p>
<h3 id="反向传播">反向传播</h3>
<p>方向传播是计算损失函数的梯度，即$\Delta w=\alpha\frac{\partial L}{\partial w}$，$\Delta b=\alpha\frac{\partial L}{\partial b}$，$w = w - \Delta w $，$b = b - \Delta b $，其中$\alpha$是学习率。根据微积分中的<strong>梯度方向是函数值增长方向</strong>的原理，通过向梯度方向的反方向更新权重与偏置，来使得损失函数的取值最小化的过程。</p>
<p>s</p>
<h2 id="代码">代码</h2>
<p>以下采用简单的一元线性函数$y=wx+b$进行实验，实现感知器。激活函数选择线性函数，损失函数采用平方误差函数$Loss(\hat y,y)=(\hat y-y)^2$，平方误差函数对w的偏导数为$\frac{\partial L}{\partial w}=2(\hat y-y)\times x$，对b的偏导数为$\frac{\partial L}{\partial b}=2(\hat y-y)$。本段代码大约在30epoch后收敛。</p>
<h3 id="pure-python">Pure Python</h3>
<pre><code class="language-python">import numpy as np

# generate the data and add random noise
a = 3
b = 1
data_size = 1000
train_x = np.random.randn(1, data_size)
train_y = a * train_x + b + 0.1 * np.random.randn(1, data_size)


class Network:
    def __init__(self, size, learning_rate):
        self.w = np.random.randn(size)
        self.b = np.random.randn(1)
        self.learning_rate = learning_rate

    def y_hat(self, x):
        return np.dot(self.w, x)+self.b

    def loss(self, x, y):
        return np.square(self.y_hat(x) - y)

    def update(self, x, y):
        gradient = 2*(self.y_hat(x) - y)
        self.w -= gradient*self.learning_rate*x
        self.b -= gradient*self.learning_rate

    def train(self, x, y):
        for index in range(y.shape[1]):
            self.update(x[:,index], y[:,index])


network = Network(1, 1e-4)

valid_x = np.random.randn(1, data_size//3)
valid_y = a*valid_x + b + 0.1*np.random.randn(1, data_size//3)

epoch = 30
for i in range(epoch):
    print(&quot;epoch:{}/{}&quot;.format(i,epoch))
    network.train(train_x, train_y)
    print(&quot;w:{},b:{},train_loss:{},valid_loss:{}&quot;.format(network.w, network.b,
                                                         np.sum(network.loss(train_x, train_y))/data_size,
                                                         np.sum(network.loss(valid_x, valid_y))/(data_size//3)))

</code></pre>
<h3 id="mindspore">mindspore</h3>
<p>参考<a href="https://www.mindspore.cn/tutorials/zh-CN/r1.5/linear_regression.html" target="_blank">简单线性函数拟合</a>。</p>
<pre><code class="language-python">import numpy as np
import matplotlib.pyplot as plt
from mindspore import dataset as ds
from mindspore import nn
from mindspore import Tensor
from mindspore import Model


def generate_data(data_size, w=3.0, b=1.0):
    for _ in range(data_size):
        x = np.random.randn(1)
        y = w * x + b + 0.1 * np.random.randn(1)
        yield np.array([x]).astype(np.float32), np.array([y]).astype(np.float32)


def create_dataset(data_size, batch_size=16, repeat_size=1):
    input_data = ds.GeneratorDataset(list(generate_data(data_size)), column_names=['data', 'label'])
    input_data = input_data.batch(batch_size)
    input_data = input_data.repeat(repeat_size)
    return input_data


def model_display(net):
    model_params = net.trainable_params()
    for param in model_params:
        print(param, param.asnumpy())

    x_model_label = np.array([-10, 10, 0.1])
    y_model_label = (x_model_label * Tensor(model_params[0]).asnumpy()[0][0] +
                     Tensor(model_params[1]).asnumpy()[0])

    x_label, y_label = zip(*generate_data(data_number))

    plt.axis([-10, 10, -20, 25])
    plt.scatter(x_label, y_label, color=&quot;red&quot;, s=5)
    plt.plot(x_model_label, y_model_label, color=&quot;blue&quot;)
    plt.show()


class LinearNet(nn.Cell):
    def __init__(self):
        super(LinearNet, self).__init__()
        self.fc = nn.Dense(1, 1)

    def construct(self, x):
        x = self.fc(x)
        return x


data_number = 100
batch_number = 16
repeat_number = 1

ds_train = create_dataset(data_number, batch_size=batch_number, repeat_size=repeat_number)

net = LinearNet()
model_display(net)
net_loss = nn.loss.MSELoss()
opt = nn.Momentum(net.trainable_params(), learning_rate=0.005, momentum=0.9)

model = Model(net, net_loss, opt)
epoch = 10
model.train(epoch, ds_train, dataset_sink_mode=False)

for net_param in net.trainable_params():
    print(net_param, net_param.asnumpy())

model_display(net)

</code></pre>
    </div>
</article>




            </div>
            <aside class="col-12 col-md-3 float-left sidebar">
    
    <div class="sidebar-item sidebar-pages">
        <h3>页面</h3>
        <ul>
            
            <li>
                <a href="/">Home</a>
            </li>
            
            <li>
                <a href="/archives/">Archives</a>
            </li>
            
            <li>
                <a href="/about/">About</a>
            </li>
            
            <li>
                <a href="/search/">Search</a>
            </li>
            
            <li>
                <a href="/index.xml">RSS</a>
            </li>
            
        </ul>
    </div>
    
    <div class="sidebar-item sidebar-links">
        <h3>链接</h3>
        <ul>
            
            <li>
                <a href="https://github.com/ruaqy" target="_blank"><span>GitHub</span></a>
            </li>
            
            <li>
                <a href="https://gitee.com/ruqy" target="_blank"><span>Gitee</span></a>
            </li>
            
            <li>
                <a href="https://space.bilibili.com/13382902" target="_blank"><span>Bilibili</span></a>
            </li>
            
        </ul>
    </div>
    
    <div class="sidebar-item sidebar-tags">
        <h3>标签</h3>
        <div>
            
            <span>
                <a href="/tags/dl/">DL</a>
            </span>
            
            <span>
                <a href="/tags/make-up/">Make Up</a>
            </span>
            
            <span>
                <a href="/tags/matlab/">MATLAB</a>
            </span>
            
            <span>
                <a href="/tags/mindspore/">mindspore</a>
            </span>
            
            <span>
                <a href="/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/">机器学习</a>
            </span>
            
        </div>
    </div>
    <div class="sidebar-item sidebar-toc">
        <h3>目录</h3><nav id="TableOfContents">
  <ul>
    <li><a href="#设计思路">设计思路</a>
      <ul>
        <li><a href="#前向传播">前向传播</a></li>
        <li><a href="#反向传播">反向传播</a></li>
      </ul>
    </li>
    <li><a href="#代码">代码</a>
      <ul>
        <li><a href="#pure-python">Pure Python</a></li>
        <li><a href="#mindspore">mindspore</a></li>
      </ul>
    </li>
  </ul>
</nav></div>
</aside>

        </div>
        <div class="btn">
    <div class="btn-menu" id="btn-menu">
        <i class="iconfont icon-grid-sharp"></i>
    </div>
    <div class="btn-toggle-mode">
        <i class="iconfont icon-contrast-sharp"></i>
    </div>
    <div class="btn-scroll-top">
        <i class="iconfont icon-chevron-up-circle-sharp"></i>
    </div>
</div>
<aside class="sidebar-mobile" style="display: none;">
  <div class="sidebar-wrapper">
    
    <div class="sidebar-item sidebar-pages">
        <h3>页面</h3>
        <ul>
            
            <li>
                <a href="/">Home</a>
            </li>
            
            <li>
                <a href="/archives/">Archives</a>
            </li>
            
            <li>
                <a href="/about/">About</a>
            </li>
            
            <li>
                <a href="/search/">Search</a>
            </li>
            
            <li>
                <a href="/index.xml">RSS</a>
            </li>
            
        </ul>
    </div>
    
    <div class="sidebar-item sidebar-links">
        <h3>链接</h3>
        <ul>
            
            <li>
                <a href="https://github.com/ruaqy" target="_blank"><span>GitHub</span></a>
            </li>
            
            <li>
                <a href="https://gitee.com/ruqy" target="_blank"><span>Gitee</span></a>
            </li>
            
            <li>
                <a href="https://space.bilibili.com/13382902" target="_blank"><span>Bilibili</span></a>
            </li>
            
        </ul>
    </div>
    
    <div class="sidebar-item sidebar-tags">
        <h3>标签</h3>
        <div>
            
            <span>
                <a href="/tags/dl/">DL</a>
            </span>
            
            <span>
                <a href="/tags/make-up/">Make Up</a>
            </span>
            
            <span>
                <a href="/tags/matlab/">MATLAB</a>
            </span>
            
            <span>
                <a href="/tags/mindspore/">mindspore</a>
            </span>
            
            <span>
                <a href="/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/">机器学习</a>
            </span>
            
        </div>
    </div>
    
    
    
    <div class="sidebar-item sidebar-toc">
        <h3>目录</h3>
        <nav id="TableOfContents">
  <ul>
    <li><a href="#设计思路">设计思路</a>
      <ul>
        <li><a href="#前向传播">前向传播</a></li>
        <li><a href="#反向传播">反向传播</a></li>
      </ul>
    </li>
    <li><a href="#代码">代码</a>
      <ul>
        <li><a href="#pure-python">Pure Python</a></li>
        <li><a href="#mindspore">mindspore</a></li>
      </ul>
    </li>
  </ul>
</nav>
    </div>
    
    
  </div>
</aside>
    </main>

    <footer>
    <div class="container-lg clearfix">
        <div class="col-12 footer">
            
            <p>
                除特殊注明部分，本站内容采用 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/" target="_blank">CC BY-NC-SA 4.0</a> 进行许可。
            </p>
            
            <span>&copy; 2023-2023
                <a href="/">RQY</a>
                 | <a href="https://github.com/dsrkafuu/hugo-theme-fuji">Source code</a> 
                | 基于 <a href="https://github.com/dsrkafuu/hugo-theme-fuji/"
                   target="_blank">Fuji-v2</a> &amp; <a href="https://gohugo.io/"
                                                    target="_blank">Hugo</a> 构建
            </span>
        </div>
    </div>
</footer>

    
<script defer src="https://cdn.jsdelivr.net/npm/medium-zoom@1.0.6/dist/medium-zoom.min.js" integrity="sha512-N9IJRoc3LaP3NDoiGkcPa4gG94kapGpaA5Zq9/Dr04uf5TbLFU5q0o8AbRhLKUUlp8QFS2u7S+Yti0U7QtuZvQ==" crossorigin="anonymous"></script>
<script defer src="https://cdn.jsdelivr.net/npm/lazysizes@5.3.2/lazysizes.min.js" integrity="sha512-q583ppKrCRc7N5O0n2nzUiJ+suUv7Et1JGels4bXOaMFQcamPk9HjdUknZuuFjBNs7tsMuadge5k9RzdmO+1GQ==" crossorigin="anonymous"></script>
<script defer src="https://cdn.jsdelivr.net/npm/prismjs@1.27.0/components/prism-core.min.js" integrity="sha512-LCKPTo0gtJ74zCNMbWw04ltmujpzSR4oW+fgN+Y1YclhM5ZrHCZQAJE4quEodcI/G122sRhSGU2BsSRUZ2Gu3w==" crossorigin="anonymous"></script>
<script defer src="https://cdn.jsdelivr.net/npm/prismjs@1.27.0/plugins/autoloader/prism-autoloader.min.js" integrity="sha512-GP4x8UWxWyh4BMbyJGOGneiTbkrWEF5izsVJByzVLodP8CuJH/n936+yQDMJJrOPUHLgyPbLiGw2rXmdvGdXHA==" crossorigin="anonymous"></script>



<script defer src="/assets/js/fuji.min.645f1123be695831f419ab54c1bcba327325895c740014006e57070d4f3e5d6b553e929c4b46f40ea707249e9c7f7c2a446d32a39ce7319f80a34525586a8e0f.js" integrity="sha512-ZF8RI75pWDH0GatUwby6MnMliVx0ABQAblcHDU8&#43;XWtVPpKcS0b0DqcHJJ6cf3wqRG0yo5znMZ&#43;Ao0UlWGqODw=="></script>

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.15.3/dist/katex.min.css" integrity="sha512-07YhC3P4/vS5HdgGuNAAeIxb5ee//efgRNo5AGdMtqFBUPYOdQG/sDK0Nl5qNq94kdEk/Pvu8pmN4GYUeucUkw==" crossorigin="anonymous">
<script src="https://cdn.jsdelivr.net/npm/katex@0.15.3/dist/katex.min.js" integrity="sha512-aMDiFsrEV3KzAn9EHwyBRS7y1APjZWt/Z/73ukLN2Ca2KcGGzlOQFQSnfOdnEcehpwMaQ8edlDB/0cMX2GsHbg==" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/katex@0.15.3/dist/contrib/auto-render.min.js" integrity="sha512-ZA/RPrAo88DlwRnnoNVqKINnQNcWERzRK03PDaA4GIJiVZvGFIWQbdWCsUebMZfkWohnfngsDjXzU6PokO4jGw==" crossorigin="anonymous"></script>
<script>
  renderMathInElement(document.querySelector('div.content'), {
    delimiters: [
      { left: '$$', right: '$$', display: true },
      { left: '\\[', right: '\\]', display: true },
      { left: '$', right: '$', display: false },
      { left: '\\(', right: '\\)', display: false },
    ],
  });
</script>




</body>

</html>
